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ABSTRACT 

This paper reports on a study concerned vith rapid 
mental operations of the central processor as it performs tasks sucb 
as the comparison of rotated figures, mental arithmetic, sentence 
verif Icationi^ and memory scanning* The central processor is the site 
of most of the symbol manipulation that takes place in the human 
information processing system* One of the goals of the study is to 
demonstrate that the locus, duration and sequence of eye fixations 
are all closely tied to the activity of the central processor* The 
research is Intended to discover hov eye fixations are related to 
cognitive processes, the aim being to construct a theory of that 
relationship that generalizes across a number of task environments* 
The second goal of the research is to examine how the eye fixation 
data reveals the fine structure of the processor's activity in 
performing a number of cognitive tasks* (Author/CLK) 
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Abstract 



This paper presents a theoretical a-^count of the fecju^nce and duration 
of eye fixatioin durir^g a number Of simple ccsftitivc tasks, ^acn mental 
rotation, sentence vcx'ificdtion, ^nd quantitative com^^^rison* la eacJv rase, 
the eye fixation belmvior is linked to a processing mod^l for the task by 
assuming that the eye fixates the referent of the synibol being t>perated on* 
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A widely accepted view of tht husifiin inforniation processing system 
that most of thtj symbol manipulation liakes place in a central processor, 
sometime^s referred to as the active memory CNeisser, 1967), working memory 
CNewell 6 Simon, 1963), operational memory fPosner, 1967), or the immediate 
processor CNewell, 1973) » This paper i** concerned with the rapid mental 
operations of the central processor, and how they are reflected by the pat- 
tern and duration of eye fixations during a task involving visual input* 
We will examine the basic operators, parameters, and control structure of 
the central processor as it performs such tasks as the comparison of rotated 
figures CShepard § Hetzler, 1^71)^ mental arithmetic C^^arkman, 1971), sen- 
tence verification (Carpenter & Just, 1975), and memory scanning CStemberg, 
1969)* These tasks generally take less than 5 or 10 sec to complete, and 
can be decomposed inti> very rapid mental operations, often estimated to 
consume between 50 to SOO mse^ each* The goals of this paper are (1) to 
demonstrate that the locus, duration, and sequence of eye fixations are all 
closely tied to the activity of the central processor, and C2) to examine 
how the eye fixation data reveals the fine structure of the processor* s ac- 
tivity in performing a number of cognitive tasks* 

Our proposal is that the eye fixates the referent of the symbol ^cur- 
rently being processed if the referent is in view* That is, the fixation 
may reflect what is at the **top of the stack*** If a number of symbols are 
processed in a particular sequence, then their referents should be fixated 
in the same sequence, and the duration of fixation on each referent may in- 
dicate how lent the corresponding symbol is operated on* The obvious advan- 
tage of monitoring eye fixations is that the behavior within any parti- 
cular trial can potentially be decomposed into various stages whose durations 
can be directly measured • By contrast, a single response latency cannot be 
interpreted or decomposed without reference to latencies in other conditions* 



Another reason that eye fixations provide an appropria*": measure in cogni** 
tive tasks is that fixation behavior can be sampled at high densities per 
^iiL ^^me. A high sampling rate (say once every 200 msec) is necessary and 
appropilat1P<*iit}*^'n t^udyitjg wental operations whose durations range from ters 
to hundreds of msecT^f^^^C^i^Js^^ duration^ of inciividwal stages of processing 
(and hence changes in the duration) can be measured directly. The relation 
between duration of processes and sampling rate can be elucidated with an 
znalagy to time-lapse photographs of slow or rapid processes* If one is 
studying the behavior of gUciers, it is sufficient tt> take a photograph once 
every few weeks; but to study the blossoming of a flower, one might want 
hourly photographs* Similarly, to study the rapid mental operations of the 
central proce^s^or, it is desireable to monitor its behavior many times per 
trial, so as to separate the behavior into sxage.^. 'iTie trace of the stages 
may provide a specification of their respective durations and the se^tueiKie 
in vrfiich they occur. 

Eye fixation studies have their historical roots in cognitive research 
dealing with reading. Almost one hundred years ago in 1876j, Javal (cited 
by ^4ac)tworth, 1974) observed young children's eyes during reading, and con- 
trary to the then popular conception of a continuous sweep across a line of 
prints he discovered that th^ eye made a series of discrete pauses separato'i 
by jumps, While some research pursued the role of eye fixations in readJ<;Ag 
(cf* Buswell, 1922; Dearborn, 1906; Huey, 1908} Woodworth, 195S3), imicK of 
the subsequent psychological research focused on the jumps (saccades) rather 
than the pausej (cf . , Alpem, 1962; Ditchbum, 1973; Yarbus, 1967, for over- 
views)* and the behaviors that were investigated were oculo-mot'Jr rather 
than cognitive* Recently, there lias beei) renewed research in-ierest in the 
pauses themselves, and how relate to underlying cognitive processes 

(cf. , Tichomirov 6 Posnyanskaya, 19G5; Winikoff, 1967)* The current paper 
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will examine eye fixations in several situations and account for the locus* 
sequence, and duration of eye fixations in terms of their relationship tp 
underlying cognitive processes* 

We will examine only tasks in which the subject inust encode some infor- 
mation from a visual display, do some mental computations, and then pro- 
duce a response that is contingent on the outcome of the computations* These 
tasks are well structured in that the subjects' goals are clear to them and 
to the experimenter- We will not concern ourselves with tasks that require 
subjects simply to read or scan a display without any specified purpose or 
response- We will analyze speeded tasks, in which the subject is asked to 
work accurately but quickly* Then, we will show how the total response la- 
tencies can be divided into processing stages on the basis of the locus and 
sequence of fixations. 

The purpose of analyzing several tasks is to abstract the general cha- 
racteristics of the central processor as they are revealed by eye fixation 
behavior* Generally, research programs and resulting papers revolve around 
a particular task, such as mental rotation* sentence comprehension, or mem- 
ory scanning, attempting to discover or characterize the operations used in 
that particular tas>:* Our goals here are slightly different* While we sure- 
ly want to leaxn about the fine structure of the processes used in each task* 
we also wish to discover how eye fixations are related to cognitive proces- 
ses- The aim is to construct a theory of that relationship that generalizes 
across a number of task environments* 

PROCESSI^fG ROTATED FIGiJRES 

Eye fixations are intimately involved with our ability to visually en- 
code spatially distributed information* We were interested in whether eye 
fixations would also indicate how visual information is internally manipula- 
ted* We chose for our task domain the "mental rotation" studies, in which 
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people compare two figures in order to determine whether or not they depict 
the same three-dimensional object (Shepard 6 Metzler, 1971). In these stu- 
dies, subjects were timed while they decided whether two figures were views 
of the same object (Fig. la or lb), or views of different objects. The tyo 
objects in the Different trials (Fig. Ic) differed by a reflection (as well 
as by a rotation). The main independent variable was the angular disparity 
between the two views of the same object, that is, the amount of physical 
rotation necessary to align the two figures into congruence. The response 
latencies for the Same trials increased linearly with the degree of angular 
disparity. Shepard and ^fetzler attributed this increase in response time 
to a process of mental rotation. The slope of the response times as a func- 
tion of the angular disparity was postulated to reflect the rate of mental 
rotation. 

Insert Figure I about^here 

There are several key questions about the processes underlying perfor- 
mance in this task that are not easily answered by the response latency 
studies. We proposed that the following questions about the microstructure 
of the processes could be addressed by an eye fixation study. 

1. How does the subject know which parts of the figure are to be ro- 
tated into each other? Before rotating one figure into another, the 
subject must decide which parts potentially correspond to each other. 
Eye fixations may indicate how this initial decision about correspon- 
dence is trade. 

2. How does the subject know how far to rotate one of the objects? 
One possibility would be that the subject makes some estimate of the 
angular disparity, and then performs a ballistic rotation (i.e., with 
the target orientation predetermined). Alternatively, the rotation 
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Figure ICa) A pair of Same figures with 0*^ disparity; 

(b) a pair of Same figures with 180"* disparity; 

(c) a pair of Different figures with 120 "disparity*" 
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process may be monitored at various points along the way. The eye fix- 
ations may show whether the process is monitored. 

3. Once the required rotation has been performed, how does the subject 
know whether the two figures represent the same object or not? The eye 
fixation behavior may reveal the comparison process that determines whe- 
ther the two figures match or not after rotation. 

We can examine these issues in some detail, by assuming that subjects 
fixate the external referent of the internal symbol being processed. Our 
objective was to identify component >^rocesses in this task by analyzing the 
scan paths and by observin^^ how they changed with angular disparity. In a 
pilot eye-fixation experincnt subjects compared tivo figures with different 
orientations in the picture plane, tiie plane perpendicular to the subjects* 
line-of-sight. The results suggested that there were three stages in the 
processing that we will call (1) search, C2)transformation and comparison, 
and C3)confirmation. 

In the first stage, there is a search for segments of the two figures 
that Superficially correspond to each other, for example, two segments at 
the end of the figures that both have three visible faces. The function of 
the search process is to select segments of the two figures that can poten- 
tially be transformed one into the other. During the next stage, transfor- 
mation and comparison, the two corresponding segments are rotated into each 
other. A transform-and-comparc operation is applied stepwise to the repre- 
sentations of the two segments. Each step of the transformation may corres- 
pond to a rotation, such that at the end of the transformation, the segment 
is represented at a new orientation. Each step of the transformation is fol- 
lowed by a comparison to drtoiviine whether the two orientations are now con- 
gruent. This stepwise transform-and-compare process continues until the 

o 10 
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necessary number of transformations have been made to make the internal re- 
presentations of the two segments sufficiently congruent in orientation. The 
third stage, confirmation, involves a check of whether the rotation that 
brought the two segments into congruence wiJl also bring other portions of 
the two figures into congruence. Processes roughly similar to search, trans- 
formation, and confirmation have been suggested by Metzler and Shepard^ and 
their subjects' introspective reports supported the suggestions (1974, p. 169 
and 178) . The eye fixation data allow us to separate the performance into 
the three stages, and specify the nature of the processing within each stage. 

Method . The experiment was a Same-Different task in which the subject 
was timed and her eye fixations recorded while she decided whether two figures 
depicted the same object or two objects that were mirror images of each oth- 
er. The stimuli were three drawings shown on the left-hand side in Figure 1 
plus their mirror images, for a total of six basic figures. In the Same tri* 
als, the left-hand figure could be rotated clockwise 180* or less In the pic- 
ture plane to bring it into complete congruence with the right-hand figure. 
The amount of rotation necessary to bring the two figures into congruence 
varied from 0* to 180** in steps of 20**, for a total of 10 possible angular 
disparities. To construct a Different pair, the right-hand figure of a Same 
pair was replaced by its mirror image isomer. The mirror image figure was 
constructed by reflecting the original figure through a plane in three-dimen- 
sional space (see Metzler 5 Shepard, 1974). There was a Same and a Different 
pair for each of the 6 basic figures at each of the 10 angular disparities^ 
for a total of 120 pairs of stimulus figures. 

The two figures were displayed side by side, with the left-hand figure 
randomly assigned to one of three orientations. The center-to-center distance 
bct;%ecn the fijiurcf; v;zs 15*5 cm, and each figure was between 10 and 10. S cm 
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wide* The stimulus pairs were displayed on a standard video monitor at a 
viewing distance that was set for each subject at between S3 and 68 cia so 
so as to keep the horizontal £tnd vertical excursion of the eye spot constant. 
Thus each figure subtended about 10** of visial angle and the center-to-cen- 
ter angle was about IS**, depending on the viewing distance. 

The eye spot was electronically superimposed on a picture of the stiinu- 
lus display, and the composite was recorded on videotape for later scoring* 
Subjects initiated a trial by fixating a point in the middle of the left- 
hand side of the screen and pushing a "ready" button* The eye spot was cal- 
ibrated with respect to this fJxation point before each trial* After cali- 
bration, the fixation point dir^iippeared and half a second later, the stimulus 
appeared* The subject respondeu Same or Different by pressing one of two 
microswitches with the index and third finger of her dominant hand* The 
stimulus presentation and timing of the response were controlled by a DDP-116 
computer* Head movements were minimized by using a bite bar and head rest. 
The 120 stimuli were presented in a random order, and distributed over two 
testing sessions, separated by at least one day. The subjects received 60 
practice trials before the experiment began. The three paid subjects were 
right-handed females of college age, with 20-20 corrected vision. Five other 
subjects were eliminated because they made more than 15% errors during the 
60 practice trials* 

The locus of the eye 5pot, relative to the ten cubes that made up each 
figure, was scored on each frame of the videotape, namely once every 16*7 
msec* When the eye spot was located on the same cubes in a sequence of suc- 
cessive video frames for at least 100 msec, the frames were aggregated into 
a single fixation* The occasional corrective eye movement that occurs after 
a saccade (cf. Rartz, 1967) rosnlted in some very short fixations* These 
were aggregated into the main fixation if the fixations were on adjacent cubes 
Q of the figure* 
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Results . The mean response latencies for correct Same trials, shown in 
Figure 2, increased monotonically with increaising angular disparity. All 
three subjects showed a linear increase between 0^ and 100**, but the curves 
were positively accelerated beyond 100**. Thr mean latencies from 0*^ to ICO*^ 
disparity have a pattern similar to that obtained by Metsler and Shepard (1974). 

Eye fixation results . One striking feature of the eye fixation beha- 
vior was that subjects systematically looked back and forth between the left 
and right figure. 2 por example, at 0** disparity, subjects initially fixated 
the left figure, then looked over at the right hand figure, then looked back 
at the left, and frequently looked back at the right hand figure for a second 
time, for a total of three switches between the two figures. The mean number 
of such switches between figures increased with angular disparity, as shown 
in Figure 3* 

Insert Figure^S^about^here 

The next step of the analysis was designed to determine exactly what 
subjects were looking at, and how the pattern of their fixations might re- 
veal the microstructure of the underlying cognitive operations. To classi- 
fy the locus of the eye fixation, we divided each figure into three main 
segments: the arm whose third face of the end cube was visible (open), the 
arm whose third face of the end cube was not visible (closed)* and a central 
angle. For example, in Figure la, the upper arm will be called the open arm, 
while the lower arm will be called the closed, and the four central cubes 
constitute the central angle. The locus of the eye spot was scored according 
to the locus of its centroH with respect to the three segments* 

The simplest way to di^iJcribe our scoring procedure is to apply it to a 
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Figure 2. Mean response latency for Same trials as a function of 
angular disparity for the 3 subjects. 
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Figure 3. Mean number of switches for Same trials 
as a function of angular disparity. 
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few representative scan paths. Figure 4 shows a scan path for a Same trial 
with 0^ disparity. After the initial fixation on the center of the left fi- 
gure, the subject fixated corresponding closed arms at the upper part of 
each figure. Then the open arms at the bottom of each figure were fixatec^. 

_Il?55r5»!;i2ure 4_about Jiere_ 

To make our analysis of the scan paths precise, we constructed rules 
for classifying instances of search, transformation and comparison, and con- 
firmation. The most prominent property of the scan paths was that the sub- 
ject would repeatedly look back and forth between corresponding segments of 
the two figures without any intervening fixations on other segments. We iden- 
tified the repeated fixation of corresponding segments with the transforma- 
tion and comparison process, l^en the same pair of segments was involved in 
two transformation episodes separated by other fixations, their durations 
were combined. The transformation and comparison process is evident in fix- 
ations 5 to 8 of the scan path shown in Figure 5, where the figures have an 
80* disparity. In fixations 5 to 8, the subject looked back and forth be- 
tween the closed arms of the two figures, for a total of 1185 msec. 

Insert Figure 5 about here 

We identified the search process with the initial portion of the scan 
path that preceded the first instance of transformation. Applying these 
rules to the scan path in Figure 5, fixations 1 to 4 would be attributed to 
search, for a total of 818 msec. In Figure 4, where the angular disparity is 
much smaller, the duration of the search process (351 msec, fixation 1) is 
much shorter. 

We identified the third process* confirmation, as a short sequence of 
fixations between corresponding parts of the two figures other than the trans- 
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Figure 4* The figure indicates the sequence of fixations on a correct 

Same trial in which the disparity was 0*** The subject's total 
response latency was 1296 msec, of which 11% had no visible eye 
spot. The locus and duration of the fixations are as follows: 



Fixation 


Figure 


Location 


Duration 


1. 


Left 


Center 


351 msec 


2. 




Closed arm 


150 msec 


3. 


Right 


Closed arm 


200 msec 


4. 




Open arm 


200 msec 


S. 


Left 


Open arm 


250 msec 
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The figure indicates the sequence of fixations on a correct 
Same trial in which the disparity was 80**. The subject*s 
total response latency was 3574 msec^ of which 9% had no 
visible eye spot. The locus and duration of the fixations 
are as follows : 



Fixation 


Figure 


Location 


Duration 


1. 


Left 


Center 


200 msec 


2. 




Center 


301 msec 


3. 


Right 


Open arm 


167 msec 


4. 




Center 


150 msec 


S. 




Closed aim 


167 msec 


6. 


Left 


Closed aim 


200 msec 


7. 




Closed arm 


317 msec 


8* 


Rigjit 


Closed arm 


501 msec 


9. 


Left 


Center 


250 msec 


10. 




Open am 


200 msec 


11. 


Right 


Center 


484 msec 


12. 




Open arm 


317 msec 
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formed segments. Confirmation could appear as a scan from the central angle 
to an arm on one figure, then a similar scan on the other figure. Figure S 
shows an example of the confirmation process where the fixations proceed from 
the central angle to the open aim on the left figure (fixations 9 and 10) and 
then a similar scan is executed on the right (fixations 11 and 12). In the 
scan path in Figure 4, the last two fixations on the open aims (fixations 4 
and 5) also exemplify confirmation. 

To see how well the model fits the eye fixation data, we scored the scan 
paths of 100 of the 171 correct Same trials. Seventy-one trials were not 
scored because of apparatus failure, in which the optical system failed to 
capture an eye spot that was visible at least 85% of the time. The mean re- 
sponse latencies from the 100 trial sample are very similar to the data for 
all 171 correct Sajne trials, as shown in Figure 6, so the sample appears to 
be representative. 

^Insert Figure 6 about here 

The analysis of the scan paths allows us to examine how the total pro- 
cessing time is distributed across search, transformation and comparisons^ 
and confirmation stages as a function of angular disparity* As Figure 7a 
shows » the time spent initially searching tfi'e figures increased with angular 
disparity^ from about 300 msec at 0** to about 1600 msec at 180*. The bulk 
of the processing time was spent in transfoxaation and comparison, as shown 
in Figure 7b* The duration of this stage increased markedly with increasing 
angular disparity, from about 500 msec at 0*" to 3800 msec at 180** The aver- 
age time spent in the third stage, conf ixmation, increased from 450 msec at 
0* to 2300 msec at 180*, as shown in Figure 7c* Thus, for a typical trial, 
say at 80* disparity, 21% of the time was consumed by initial search, 39% by 
transformation and comparison, and 26% by confirmation* The remaining 14% 
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Figure 6. Mean response latency as a function of angular disparity for 

all correct Same trials and for 100 correct Same trials In which 
eye fixations were scored* 
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was distributed between saccades (about 10%) and remaining **other" fixations 
(about 4%) that did not fit any of the three categories. 

^Insert Figure 7^atout^here 

Even though the duration of transformation did not increase linearly, 
the total response time did so at least between 0** and 100*. It is easy to 
see in Figure ^ (panels a through e) how the durations of several monotone 
functions cumulate to produce a linear increase in total latencies between 
0** and 100*^, shown in Figure 6. Some of the nori- linearity beyond 100^ is 
unaccounted for, even by the eye fixation analysis, and has been classified 
as '"other" eye fixation behavior. This miscellaneous behavior may result 
from the subject losing track of her place in the process, or attempting to 
start over. Occurrences of this behavior inflated the total response times, 
particularly at 180**. 

We mentioned earlier that a striking feature of the scan paths was that 
the subject repeatedly looked back and forth between the two figures, and 
the number of such switches between the figures increased with angular dis* 
parity. We measured how these switches were distributed across the three 
stages of processing, as shown in Table 1. The number of switches associa- 
ted with the search stage remained quite low (usually one or less) at all 
disparities. Most of the switches occurred during the transform and coiiq?are 
process, during which the number of switches increased monotonlcally with 
angular disparity. The switching data from this stage will play a key role 
in the development of our model. Finally, trie switches during confirmation 
increased with angular disparity, but not as much as for transformation. 
The classification procedure also categorizes the switches that occur during 
the transition from one stage to another, and the number of such switches re- 
mains fairly constant across angular disparities. 



2000 
1000 

4000 
3000 
2000 

- 1000 

O 

< 2000 
Q 1000 



_ a. 



1000 



1000 



b. 



9 ^ 
J L 



^ 0* 



- d. 



- e. 



SEARCH 



J I I L 



J I ■ ' 



TRANSFORMATION 
AND COMPARISON 



CONFIRMATION 



_L- 



OTHER 



_i_ 



9 



SACCADES 



1 f t. 



J I I ! I L 



9 



J 1. 



J I 1 I L 



0 40 80 120 160 
ANGULAR DISPARITY (degrees) 



Figure 7. 
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Mean duration of various processing stages in Same trials as a 
function of angular disparity* 



22 



12* 



^Insert Table^l^abouthere^^ 

As might be expected, the average number of fixations increased with 
angular disparity, from 6 fixations at 0**, to 31 at 180*** Also, the average 
duration of a fixation increased from 200 msec at 0*^ to over 320 msec at 180*- 

Incorrect Same trials . Error trials have often been ignored by chron- 
ometric models of cognitive processes because it is difficult to attribute 
errors to a particular stage of processing (one exception is the work on the 
speed-accuracy trade-off, cf*, Wickelgren, 1974). An incorrect response in 
the rotation task could result from an error during any one of the stages 
of searching, transfomdng and comparing, confirming, or in executing the 
final motor response- An example of a transformation error would be to ro- 
tate a segment about the wrong axis and incorrectly conclude that two Same 
figures represent different objects. The total response latency alone pro- 
vides insufficient information to localise the error on a particular trial 
to a particular stage. However, the eye fixations do provide clues about 
the reasons for some of the errors. There was a total of nine errors on the 
Same trials^ all on angular disparities greater than 120*. On five of the 
nine trials, the subject attempted to transform non -corresponding segments. 
That is^ the initial search process selected two segments that were in fact 
not corresponding. The subsequent transformation and confirmation stages 
failed to detect this error. The scan path in Figure 8 demonstrates this 
type of error in which the subject erroneously selected the open arm on the 
left and closed arm on the right as corresponding^ then looked back and fcrth 
between them in fixations 2 to 8 and 10 to 12, and attempted confirmation in 
fixations 13 to 21. 

Infcrt I'ifiure 8 about hero 
In the remaining four error trials, subjects did successfully complete 
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Table 1 







Distribution 


of switches in 


100 trial sample 










Mean Number 


of Switches during: 






Angular 


Initial 


Transformation 




1 X cin^ X tXUIl 


Switches Not 




Disparity 


Search 


and Comparison 






Accounted For 


Total 


0* 


0.0 


1.0 


n Q 

V ■ 7 


V ■ u 


0.2 


2.7 


20* 


0.4 


1.1 


0.9 


0.8 


0.1 


3.3 


40* 


0.2 


1.7 


1.2 


0.9 


0.2 


4.2 


60* 


0.3 


1.7 


1.5 


0.7 


0.4 


4.6 


80' 


1.0 


3.0 


1.5 


0.4 


0.3 


6.2 


100* 


0.5 


2.7 


1.5 


0.8 


0.5 


6.2 


120* 


1.1 


2.9 


2.1 


0.6 


1.2 


7.9 


140* 


2.2 


3.6 


1.8 


0.7 


0.7 


8.9 


160* 


1.3 


4.0 


2.2 


0.8 


0.9 


9.2 


180* 


1.6 


5.7 


2.3 


0.8 


1.8 


12.2 
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The figure indicates the sequence of fixations on an incorrect 
Same trial in which the disparity was 140**. The subject*s to- 
tal response latency was 8S67 msec, of which 13% had no visible 
eye spot* The locus and duration of the fixations are as follows; 



Fixation 


Figure 


Location 




1* 


Left 


Center 




2 




Open arm 


134 msec 


3. 


Right 


Closed arm 


200 jnsec 


4. 




Closed arm 


200 msec 


S. 


Left 


Open arm 


468 i&sec 


6. 




Open arm 


317 msec 


7. 


Right 


Closed arm 


200 msec 


8. 




Closed arm 


334 msec 


9. 


Left 


Center 


334 msec 


10. 




Open arm 


117 msec 


11. 


Rigjit 


Closed arm 


401 msec 


12. 


Left 


Closed arm 


150 msec 


13. 


Right 


Center 


150 msec 


14. 


Left 


Center 


418 msec 


IS. 


Right 


Closed arm 


2Zl msec 


16. 




Center 


200 msec 


17. 




Center 


568 msec 


18. 


Left 


Open arm 


768 msec 


19. 


Right 


Center 


450 msec 


20. 




Center 


902 msec 


21. 




Center 


53 A '^'^r^'^. 
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the initial search process and subsequent fixations alternated between corres- 
ponding segments of the two figures. This suggests that the source of the er- 
ror must have occurred in some subsequent stage such as the transformation, 
confirmation, or response execution. 

Different trials . The response latencies for Different trials were 
long (an average of 4 sec longer than Same trials) and variable* The angu- 
lar disparity between two figures is not really well defined for a Different 
trial, since the two figures cannot be physically rotated into congruence* 
The total response latencies alone gi^e no indication of how processing time 
was distributed across the three stages. However, the pattern of eye fixa- 
tions allow us to follow the sequence of processing stages and to determine 
which stages consume the extra 4 sec of processing* 

The scan paths indicate that the initial search process in Different 
trials starts out similarly to Same trials. However, in a Different trial, 
the segments selected by the search stage cannot be in coinplete correspond- 
ence. For exan^le^ in the Different pair shown in Figtire ic, the short arm 
in the left figure is closed while the short arm in the right figure is open. 
So subjects must select a pair on the basis of length or openness. In all 
seven Different trials involving stimulus pair Ic that we analyzed^, the ini- 
tial selection was based on the feature of length. In Different trials in- 
volving the objects depicted in Figures la and Ib^, the two open arms have 
the same length, but differ in the way they are joined to the center. In 
two of the three analyzed trials involving these objects, the transformation 
was between open arms. In the third case, it was between arms that were 
similarly joined to the center. 

The confirmation process is extremely important in the Different trials. 
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since it leads to the discovery that the inter-segment relations are not the 
same in the two figures and hence that the figures are different. In fact» 
one of the most prominent features of the Different scan paths is the large 
amount of confirmation behavior that they contain* In the ten analyzed DJ.f- 
ferent trials* the confirmation process consumed an average of 4195 msec, or 
49% of the total duration- 

The prolongation of the confirmation process is not the only reason for 
the very long response latencies for Different trials ► On some trials, af- 
ter going through a complete search-transform-and-unsuccessfully-conf irm se- 
quence, subjects make a second attempt at searching, transforming and con- 
firming a different pair of segments. Occasionally, a lengthy search 
stage involved an exajnination of all the possible ways of pairing the seg- 
ments, and that kind of search led directly to a response of Different, with- 
out any transformation* Thus, the durations of all three stages increased 
during Different trials, but the duration of confirmation increased the most. 

One scan path that exemplifies the processing on Different trials is 
shown in Figure 9. Fixations 1 to 4 reflect the initial search for corres- 
ponding segments, consuming 1436 msec. Then, there is a transformation and 
comparison of the short arms of each figure in fixations 5, 6, and 7, consu- 
ming 919 msec. Fixations 8 to 15 reflect the confirmation process, consu- 
ming 3l75 msec. We presume that it is during confirmation that the subject 
determined that the relation between the arm and central angles was differ- 
ent in the two figures. In this trial* the bulk of the processing time was 
consumed by the confirmation stage. 

Insert Figure 9 about here 



Figure 9. The figure indicates the sequence of fixations on a correct Dif- 
ferent trial* The subject's total response latency was S868 msec 
of which 6% had no visible eye spot* The locus and duration of 
the fixations are as follows: 



Fixation 


Figure 


Location 


Duration 


1. 


Left 


Center 


234 msec 


2* 




Center 


S18 msec 


3. 


Right 


Closed arm 


367 msec 


4. 


Left 


Center 


317 msec 


S. 




Closed arm 


184 msec 


6* 


Right 


Open arm 


434 msec 


7. 


Left 


Closed arm 


301 msec 


8* 




Center 


2S1 msec 


9* 




Closed arm 


217 msec 


10* 


Right 


Open arm 


63 S msec 


a. 




Center 


S18 msec 


12* 




Center 


234 msec 


13* 


Left 


Closed arm 


S8S msec 


14* 




Center 


167 msec 


IS* 


Right 


Center 


568 msec 
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The Processing Model of the Rotation Task 

The internal representation . We propose that the processor operates 
on one segment of the figure at a time, and that the representation of the 
segment is scheriatic. The representation mu*t include information about 
the segment's absolute orientation in space, as well as some defining fea- 
ture such as its length or whether it is a closed or open arm. This infor- 
mation can be efficiently represented as the vector formed by the major 
axis of the segment. Moreover, if the vector has its initial point at the 
origin of the reference frame, then the segment can be represented by the 
spherical coordinates of the end point of the vector. For example, an open 
arm might be represented (OPEN (r, e» (fr)) where r is the length of the seg- 
ment, and e and (j) define the orientation of the segment. 

Thft initial search process . The scan paths indicate that the search for 
corresponding segments uses a simple heuristic. Once a segment of one figure 
has been identified, then the search for the corresponding segment starts 
in the corresponding location of the other field* For example, if the long 
arm is in the upper-right hand corner of the left field, then the search for 
the corresponding segment begins in the upper-right hand of the right field* 
If there is no segment in the upper right, then the segment 
nearest the upper right is examined, ^Th^ duration of this search Pro- 
cess increases with angular disparity for two reasons. First, with increa- 
sing disparity, the corresponding segments are in successively more dissim- 
ilar locations* At 0** disparity, corresponding segments have identical lo- 
cations in their respective fields. However, as the disparity increases 
from 0*^, absolute location is a successively poorer cue for finding corres- 
ponding segments, and the heuristic must be suppleipented by an active search. 
The scconti reason for the increase is that at larger disparities, the prob- 
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ability of selecting and attempting to transform non-corresponding segments 
increases and this incorrect transformation is counted as part of initial 
search* Figure 8 shows an example of the search process selecting non-cor- 
responding segjnents that are both at the top of their respective fields* 
In this trial, the incorrect search led to an error. On other trials, the 
incorrect selection of a pair of segments was detected after some transform 
mation had been attempted. Thus^ the eye fixations allow us to trace the 
initial search for corresponding features and to determine the reason for the 
increase in the duration of the search process with angular disparity* 

The transformation and comparison process i The eye fixation data also 
allow us to formulate a precise model of the transformation process. We 
propose that rotations are executed and monitored in discrete steps of ap- 
proximately 50**. The estimate of the SO** step size is based on the result 
that there is one additional switch during the transformation stage for each 
additional increment of 50^ in angular disparity, as shown in Figure 10. A 
transformation may consist of applying fi rotation rule that alters the rep- 
resentation of the orientation of a segment by 50*. For exaii5>le, an open arm 
represented as (OPEN (r, 0, might be transformed into (OPEN (r; 0 + 50** ♦)) 
It is assumed that the representations of the two segments are rotated towards 
each other by applying the 50* rotation rules first to one segment and then to 
the other, until they are within 25* of each other. This form of representa- 
tion and transformation does not impose any great computational burden, in con- 
trast to a truly analogue, holistic representation of the entire figure rotated 
by a parallel computation of the position of all its points* 

^Insert Figure^lO about here 

This model of the ti'ansformation stage is most easily cxplain<?d by wor- 
king through an example, say when two Same figures have an angular disparity 
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Figure 10. Mean number of observed and estimated switches during the 
transformation and conparison stage in Same trials as a 
fiinction of angular disparity. 
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of 80^* Suppose that the subject has encoded a particular segment of the 
left-hand figure. The first switch occurs when she fixates and encodes the 
corresponding segment of the right figure. Then, the orientations of the 
two segments are compared. The orientations differ by more than 25**; there* 
fore, she rotates her representation of the right-hand figure by 50^ counter- 
cloclcwise. After this transformation, she retrieves the representation of 
the segment on the left. In doing this retrieval, she switches her fixation 
to that figure. After it is retrieved^ she compares the two orientations* 
They would still be more than 25** apart. Therefore, she transforms the rep- 
resentation of the orientation of the left-hand figure by 50** clockwise. 
Then she retrieves the representation of the segment on the right in ordei 
to compare the two again. In doing this retrieval, she switches fixation 
over to the right figure. After it is retrieved, she con^ares the two or- 
ientations. At last^ after three switches and two applications of the rota- 
tion rule, the two segments are represented at fairly similar orientations 
(within 25* of each other). The subject would then continue on to the con- 
firmation process. 

This model can be easily summarized. After the subject has initially 
encoded one segment on a figure, sl^e proceeds through the following steps: 

1* Retrieve the represeatation of the corresponding 
segment on the other -.^gure (and switch fixation to 
other figure) . 

2* Compare the two orientations. Are they less then 
25* apart? 

a. Mo. Transform the currently fixated segment by 
50* in the direction of the other figure. Go 
back to Step I. 

b. Yes. Go on to the confirmation stage. 
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The model assumes a very close relationship between eye fixations and 
mental operations during the transformation process. The rotation rule is 
always applied to the arm that is being fixated. Applying a rotation rule 
to the representation of one arm causes the representation of the other am 
to be pushed down in the short-term memory stack. IVhen the representation 
is being retrieved to the top of the stack, the other arm is fixated anew 
(Step 1). According to this model, the number of switches during transfor- 
mation should increase monotonically with the angular disparity, but the 
increase should be in the form of a particular step function. There ^hould 
be One switch between 0^ and 25"*, two switches between 25"* and 75*, three 
switches between 75"* and 125**, and so on. Figure 10 shows that this esti- 
mated number of switches corresponds very closely to the observed number of 
switches. The increase in switches is similar to the pattern obtained for 
the duration of the transformation process, shown in Figure 7b. The data 
in Figure 7b suggest that each step of the transform and compare process 
talces about 800 msec. Thus, the model of the transformation stage gives a 
good account of the data. 

The confirmation process . Rotating two segments into similar orienta- 
tions during the transformation stage is not sufficient for deciding whether 
the figures are the Same or Different. Therefore, the third stage, confir- 
mation, determines whether segments other than the transformed ones corres- 
pond to each other. The scan paths indicated two methods for confirming 
such correspondence. One method applies the same sequence of rotation rules 
used in the transformation stage to another pair of segjnents. If this second 
rotation is successful, then the two figures are the Same. This method^* 
used on about half the trials, was identified by switches of fixation between 
a pair of segments otlicr than the transformed pair. A second method encodes 
the relation between the central angle and an arm of each figure and deter^ 
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mines whether that relation is the same in both figures* This methodtUSed 
on the other half of the trials, was identified by a scan from the center to 
the arm of one figure and then a similar scan of the other figure (see Fig- 
ure 5 for an example)* The combination of these two methods would produce a 
confirmation duration that increases with angular disparity, but the increase 
. would be at a slower rate than for the transformation duration (shown in Fig- 
ure 7)* Either method of confirmation can determine the response of Same or 
Different* 

Discussion . The eye fixation data lead to a detailed model of the pro- 
cessing in the Shepard-Metzler task, but there are (^estions about the gener<* 
alizeability of the model* Without examining a broad range of experimental 
situations, there is no way of knowing which aspects of the model are invar- 
iants of the human processing system and which aspects are task-induced* 
Consider the proposed 50^ rotation steps* It is possible that the 50^ steps 
are fundamental and invariant over tasks* Alternatively, people may be able 
to tune the size of the rotation step to the particular grain and range of 
orientation differences they are faced with in an experiment* This is a 
clear empirical question of whether the rotation operation adapts itself to 
the task environment* Similarly, one can consider whether the representa* 
tion of the figures is the same in all rotation tasks* The representations 
proposed in the current model are highly schematic, but they do contain suf** 
ficient information to perform the task* The representations might be more 
complex in tasks that demand that more information be encoded ftcmi the fig- 
ures* Just as eye fixation analyses led to a precise model for the Shepaxd-. 
Metzler task^ this methodology should also distinguish the invariant £rom^ the 
transient processes, and so lead to a general theory of mental rotation* 

Mental rotation has been described as an analogue process (Metzlet 5 
Shepard, 1974) in the sense that rotating a representation ftom one orien- 
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tation to another requires that the representation pass through some internal 
states that correspond to intermediate orientations. Although Metzler and 
Shepard (1974) "...are willing to concede that sufficiently small differences 
may be handled in discrete jumps/' the 50** step that we propose is rather 
large. It is possible that within each 50^ step there are intermediate sta- 
ges corresponding to intermediate orientations. But even with 50* steps, a 
150* rotation involves intermediate stages corresponding to 50* and 100* ro- 
tations, and so the proposed process is in a loose sense analogue. 

In summary, the scan paths enabled us to separate the processing into 
search, transformation, and confirmation stages and to hieasure the duration 
of each stage. Switches in fixation during the transformation stage indica- 
ted that the rotation was monitored in steps of approximately 50*- This 
analysis was applicable not only to the correct Same trials, but also pro- 
vided evidence on error trials and Different trials. The research shows 
how eye fixations can reveal the sequence of mental operations during the 
processing of spatial information. 

COMPARING SENTENCES WITH P];CTURES 

Under the assumption that visual gaze reflects underlying mental pro- 
cesses, we can study how people verify whether a sentence is true or false 
of an accompanying picture. In particular, we can determine how the total 
processing time is allocated among the various stages of verification. Reac- 
tion-time studies of sentence verification show that people make more errors 
and take longer to respond when verifying a negative sentence. The extra 
processing time for a negative lies between 300 and 1200 msec, depending oa 
the linguistic structure of the negative sentence (Carpenter § Just, 1975). 
The processing stages involved in verification include reading the sentence 
and internally representing it, coding the picture, and comparing the rep- 
resentation of the picture to the information from the sentence (Carpenter 



21* 



6 Justp 1975; Chase H Clark, 1972; Clark § Chase, 1972; Trabasso, Rollins 
§ Shaughnessy, 1971)* The extra processing time for negative sentences 
could be consumed in any of these stages* One purpose of the following stu^ 
dy was to monitor eye fixations in order to determine which stage of proces- 
sing consumes the extra time due to negation* 

Elsewhere, we have developed a model of how the information in a sen- 
tence might be represented and compared to information from a picture in 
sentence verification tasks (Carpenter § Just, 1975)* The model suggests 
that elements in the sentence representation are compared sequentially to 
elements encoded from the picture* Mismatches between elements result in 
additional comparisons, thereby consuming additional processing time* The 
model postulates that because of the number and nature of the mismatches, 
the number of comparison operations increases linearly from the case of 
true affirmative sentences, to false affirmatives, to false negatives^ to 
true negatives* In fact, the verification latencies in a number of studies 
have been found to increase linearly — corresponding to the increasing num^ 
ber of postulated comparisons* The current experiment examined which parts 
of the display were fixated longer during the conditions with longer response 
latencies* The primary goal was to determine how the location and duration 
of the gaze was related to the proposed stages of processing* 

An important innovation in the current methodology was that the display 
was made contingent on the locus of the gaze. The only part of the display 
(either the sentence or the picture) that was visible to the subject was 
the part at tho locus of the gaze, as depicted in the schematic diagram in 
Figure 11. This gaze- contingent display creates a functional **tunnel vision" 
in the subject by eliminating all peripheral information relevant to the 
true-false decision* The subject could not encode new information unless 
he looked at the relevant position in the display* 
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Insert Figure 11 about here 

Method * The sentences in the experiment were either affirmative, like 
Is North , or negative, Isn^t North , and involved one of the four directions, 
North, South* East, or West. The subject was told the phrase always referred 
to the location of a plus and to consider it to mean " The plus is North" or 
" The plus isnH North -" The picture contained a plus at one of the four 
compass directions, and a star at the other three. (Any one of these cha- 
racters, as well as the sentence, was displayed only when the subject direct- 
ly fixated it). When an affirmative sentence was true, or a negative sen- 
tence was false, the plus was at the place specified by the directional term 
in the sentence. In the false affirmative and true negative cases, the 
plus could have been at any one of the three remaining locations. This de- 
sign was adopted to discourage subjects from recoding negatives like IsnH 
North into corresponding affirmatives, like Is South . The analysis, however, 
is concerned only with the cases where the plus was located on the same axis* 
as the directional term in the sentence. The sentence, centered on the vi- 
deo monitor, was 5.6 mm high and 45 mm wide (50 mm for negative sentences). 
The plus and stars were 5 mm by 5 mm, and they were at a distance of 75 mm 
from the center of the screen. Subject's eyes were 64 cm from the monitor, 
on average; however, the distance was adjusted for each subject to keep the 
excursion of the eye spot constant. 

For scoring purposes, the viewing field was divided into an imaginary 
three-by- three grid, such that the sentence was located in the center square, 
while the stars and plus were in the middle top, middle bottom, middle left, 
or middle right squares. Any single fixation or sequence of fixations on one 
of these squares was scored os gaze on that location. During a trial, the 
digitizer determined the locus of the eye ^ot every 16 msec. Sixteen msec 
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Figure 11(a) Schematic diagram of the visual display In the sentence 

verification task when the eye spot (denoted by black spot) 
is on the sentence (not to scale); 
(b) visual display within the same trial when the eye spot Is 
in the North location. 
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after the eye spot was first detected in a square, the stinrulus material 
for that square appeared on the screen < As soon as the ©y© spot moved from 
that square, the stimulus was replaced by a place holder. The place holders 
for the sentence location were random letters like "ZGK BXVQ". The place 
holder for each star and plus was a dollar sign, The rapidity of the 

replacement, within 16 msec after the initial fixation of a square, made it 
relatively unobtrusive. The place holders assured that the information in 
the periphery was only locative in nature, providing a marker of where the 
subject could look to get information. 

Half a second after the subject fixated a target in the center of the 
display field and pressed a "ready" button, the sentence appeared at the 
central fixation place. The subject was timed from the onset of the display 
until his response terminated the trial* Each of the 12 subjects had 15 
practice trials and two blocks of 48 test trials. 

Results and Discussion . As Figure 12 shows, the total response times 
in the four information conditions did increase linearly from true affirma- 
tive, to false affirmative, to false negative, to true negative. In fact, 
a straight line accounts for 98.6% of the variance among the four means. 
The residual 1.4% of the variance is not significant, (2, 33)<1. Thus, 
the pattern of total latencies for the current task resembles the latency 
pattern found in other experiments (cf . , Carpenter 6 Just^ 1975). These 
analyses concern only those trials in which the subject gave a correct res- 
ponse. The frequency of incorrect responses was very Iom^ as indicated in 
Figure 12- 

Insert Figure 12 about here 

The imiiortnnt advantage of the current methodology is that the location 
and duration of the gaze allow us to break down the total response time into 
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Figure 12. Mean response times for the true affirmative (TA), false 

affirmative (FA), false negative (FN)» and true negative (TN) 
conditions. Response times are plotted as a function of the 
hypothesized number of comparison operations for each condition, 
where t( is the number of comparisons for the true affirmative 
condition. 
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finer components. For this analysis, we divided the gazes into four cate- 
gories: the initial gaze on the sentence, subsequent gazes on the sentence 
after having looked away, gazes on the location specified by the directional 
term in the sentence, and finally, gazes in pny other locations. Thus, the 
durations of all four types of gazes add up to the total response time. The 
important question was whether these durations varied systematically as a 
function of the four information conditions. 

The initial gaze on the sentence should reflect the time to read and 
represent the sentence. As Figure 13 shows, the duration of the initial 
fixation was 57 msec longer for negatives than for affirmatives, £ (1, 33) 
- 14.93, p <.0I. This result indicates that the negative sentences take 
about 57 msec longer to read and represent than the affirmatives. After 
having looked away from the sentence, subjects occasionally refixated it la- 
ter in the trial. The durations of such subsequent gazes on the sentence 
weie similar for all four information conditions, as Figure 13 shows. 

Insert Figure 13 about here 

The directional terra in the sentence can be viewed as an instruction 
for where to direct the next fixation — irrespective of whether the sentence 
was affirmative or negative. In fact^ the location specified by the direc- 
tional term was the locus of the second gaze on 92% of the trials. Subjects 
tended to fixate this location only once during a trial. The time spent ga- 
zing at this location increased linearly with the number of hypothesized 
comparison operations, tis Figure 13 shows. The straight line accounts for 
98.1% of the variance among the four means. The residual 1.9% is not sig- 
nificant^ £ (2, 33) <1. This suggests that after reading and representing 

<^ 

the sentence, the suhjcct fixated the picture and completed the comparison 

/* 

operations while fixating the location specified by the directional terra. 
The slope^ 13S msec per operation, may be interpreted as an estimate of the 
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Number* of comparisons 

Average duration spent gazing on various locations of the 
display for the true affirmative (TA), false affirmative (FA), 
false negative (FN)^ and true negative (TN) conditions. 
These coiaponents add up the total response times shown in 
Fig, 12. 
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time to compare an element from the sentence representation to one from the 
picture representation. 

Occasionally, subjects would gaze at a location other than the sentence 
or the location specified by the directional term. The frequency and dura- 
tion of these other gazes did not vary as a function of information condi- 
tion, as Figure 13 shows. 

These results indicate how the total processing time in sentence veri- 
fication is distributed among various stages. The duration of the initial 
gaze on the sentence suggests that the time needed to read and represent 
the sentence is 700 msec at most. This 700 msec enters primarily into the 
intercept of the total response time. What accounts for the difference be* 
tween the response time for the fastest condition, the 1400 msec for the 
true affirmative, and the slowest condition, the 1900 msec for the true neg- 
ative? This 500 ms>oc is consumed by the operations that compare the sen* 
tence and picture to determine their relation- In fact, these comparison 
operations are reflected in the duration of the gaze on the location speci- 
fied by the directional term. 

This analysis can tell us why negative sentences take longer to process 
than affirmatives. The. total response time was 346 msec longer for negatives 
than for affirmatives- This can be partitioned into several components. 
The largest component is the conqparison time (reflected in the duration of 
gaze at the picture) which was 267 msec longer for negatives. Secondly, 
negative sentences took 57 msec longer to read. And thirdly, subsequent 
gazes on the sentence were an insignificant 20 msec longer for negative sen* 
tences. Thus the bulk of the additional processing time for negatives is 
consumed by the operations that compare the information from the sentence to 
the picture. 
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The results show that there is a systematic correspondence between the 
mental operations and eye fixations in a sentence verification task. Under 
well-controlled conditions, the sequence of gazes on the external display 
corresponds to the sequence of mental operations in the processor. More- 
over, the duration of the gaze is proportional to the duration of the under- 
lying operations. 

QUANTITATIVE COMPARISON 

A quantitative comparison requires an order judgment Ce,g,, Which is 
larger? > or Which is brighter? , or Which is longer? ) of two or more objects 
along a common underlying dimension. The comparative judgment requires 
that the two objects be represented, and their representations be compared. 
In order to obtain more detailed evidence about the processes in this taslc, 
we devised an experiment in which subjects' eye fixations were monitored 
while they decided which of two groups of dots was larger. The response 
latencies for selecting the larger of two groups of dots strongly resemble 
the latencies for digit comparisons (Buckley § Gillman, 1974), so our task 
may produce results general izeable to digit comparisons. Furthermore, pri- 
or data (summarized by Klahr, 1973) have shown that the time to determine 
how many dots there are in a group increases monotonically from about 500 
msec for one dot, to 2200 msec for nine dots. These results suggest that 
larger groups of dots might be fixated longer if they are to be quantified. 
We hypothesized that the duration of fixation on each of the groups of dots 
might tell us how the two groups of dots were represented and processed du- 
ring a quantitative comparison task. 

Method . Subjects* eye fixations were monitored as they compared the 
sizes of two groups of dots. Each group contained from 1 to 6 dots, so there 
were IS possible pairs of unequal groups. If the word more appeared on the 
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left side of the display (as shown in Figure 14), subjects indicated whether 
the upper or the lower group contained more dots, by pressing an upper or 
lower response button. If the word was less , they judged which group con- 
tained fewer dots, A total of 60 stimuli were formed by orthogonally com- 
bining the two words, more and le ss with the IS pairs of groups and the re- 
sponses designating either the upper or the lower group. Each subject had 
four blocks of 60 stimuli, presented in a random order. A trial started 
SOO msec after the subject fixated a point at the locus of the word, and 
pressed a "ready" button. 

^Insert Figure 14^about here_ 

The computer-generated display was presented on a video monitor at a 
distance of S3 to 68 cm- The word mo re or less , 2.8 cm wide, appeared 13 cm 
to the left of the dot display* The dots formed two vertical lines one above 
the other, separated by a vertical distance of at least S cm. Each group of 
dots was .5 to 6 cm long, depending on the number of dots in the group. For. 
scoring purposes, the screen was divided into the four imaginary sectors 
indicated by the dashed lines in Figure 14. The analysis was concerned pri^ 
raarily with the distribution of the gaze across the four sectors. 

Results . The response latencies showed that this experiment replicated 
the major latency results that have been previously reported for this task 
(Buckley tt Gillman,' 1974)* The mean latencies ranged from 700 to 1100 msec. 
We will discuss the response latencies in more detail after an analysis of 
the eye fixation results. 

First of all, we examined the duration of gaze on the smaller group. 
If subjects were computing the number of dots in the group, one might ex- 
pect that the more dots there Were in the group, the longer people would 
spend looking at it. As expected, the gaze duration on the smaller group in- 
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Figure 14. Schematic diagram of the visual display in the dot 
quantitative coiiq>arison task (not to scale)* The 
dashed lines Which indicate the boundaries between 
the four sectors^ did not appear in the display* 
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creased by about 26 msec for each additional dot (see Figure IS)* This is 
within the range of subltizing rates cited by Klahf (1973), although it is 
at the low end. Thus, it is plausible that the subjects compute the number 
of dots in the smaller group. 

Insert PigPje 15 about^here 

But what about fixation on the larger group? If subjects determine the 
quantity of dots in the larger group, then gaze duration on the larger group 
should also increase with the number of dots in that group* However, Figure 
16 shows that the duration of gaze on the larger group is independent of 
the number of dots there. Thus, the two groups of dots are fixated differ- 
ently* The size of the smaller group predicts the gaze duration on the smal- 
ler group, but the size of the larger group does not predict the gaze dura- 
tion on the larger group* 

Insert FigiJJ?^16 abgut^here^ 

The proposed model * The results are consistent with a counting model 
Cc£* Parkman, 1971; Groen 6 Parkman, 1972) adapted to the dot inequality 
task. The process might start by counting one or two dots in each group, 
and checking to see if either group had been exhausted. If one group had 
been exhausted, it would be designated the smaller one. If neither had been 
exhausted, then one or two more dots might be counted in each group, and 
again there would be a check to see if either group had been exhausted* 
This process would continue until one of the groups, the smaller one^ would 
be exhausted* If the subjects were answering the question "Which group 
contains more dots?", they would simply indicate the group that had not bei&n 
exhausted. Tlie number of counts or iterations in this process would be pro- 
portional to the number of dots in the smaller group* If the gaze duration 
is proportional to the number of increments, then it follows that duration 
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Figure IS. Mean duration of gaze on the smaller grou^ of dots as a 
function of the number of dots in that group. 
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Figure 16. Mean duration of gaze on the larger group of dots as a 
function of the number of dots in that group. 
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of gaze on the smaller group should increase with the number of dots in the 
smaller group^ as it does. (This is called the min effect^ since latencies 
increase with the size of the smaller or minimum group.) Furthermore^ the 
duration of gaze on the larger group should be more or less independent of 
the number of dots in the larger group, which it is. One further prediction 
of this model is that the duration of gaze on the larger group should increase 
with the size of the smaller group. This prediction follows from the proposal 
that the dots in both groups are counted only until one group (the smaller 
one) is exhausted. This prediction is confirmed, with gaze durations on 
the larger set increasing monotonically from 160 msec when the smaller group 
contains one dot to 296 msec when the smaller group contains S dots. 

The proposed counting model requires supplementation to account for a 
persistent finding from this and previous research, namely that comparisons 
are faster when the absolute difference or "split" between the two groups is 
larger (cf. , Henmon, 1906; Johnson^ 1939 for the data on line length com- 
parisons; Buckley § Gillman, 1974; Pairbank, 1969; Moyer § Landauer^ 1967; 
Parkman^ 1971; Sekuler^ Rubin f, Armstrong, 1971 for data on digit compari- 
sons» and Buckley § Gillman, 1974 for data on dot comparisons). The split 
effect is present in both the total latencies (see Table 2) and in the gaze 
durations on both the smaller and larger groups of dots. We attribute the 
split effect to the presence of a second mechanism that can sometimes make the 
quantitative comparison by categorizing each of the two groups of dots as a 
small group or a large group. Groups of 1^ 2» or 3 dots may be classified 
as "small", whAle groups of 4, S, or 6 may be classified as "large", but the 
boundary may be variable across trials and subjects. If one group of 
dots belongs to the "small" category, and the other to the "large" category, 
then the one that belongs to the "large" category is larger. The category 
judgment mechanism may be much quicker than the counting mechanism, but it 
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would not work when the split is small, since in those cases the two groups 
would tend to belong to the same category* Pairs with large splits (splits 
of 3, 4, or S) could be processed with the quick category membership judg- 
ment much more often than pairs with small splits (1 or 2). The mean response 
latencies for any pair would be a mixture of the trials where the fast cate- 
gory membership judgment is used and trials where the counting mechanism is 
used. As the split increases, the number of fast trials contributing to the 
mean should increase, and so on average, the mean latencies should decrease. 

Insert Table 2 about here 

The two-process explanation is supported by an interaction between the 
min effect and the split effect observed in this experiment and others. 
When the split is small, the counting mechanism is more likely to be used, 
resulting in a strong min effect* That is, the response latencies increase 
with the size of the smaller group. When the split between the two groups 
is larger, the category judgment mechanism should be used more often, and so 
the min effect should decrease. The total latencies in Table 2 show this 
trend. When the split is small (namely, 1), then latencies increase by an 
average of 43 msec with each increment in the min (the smaller group). 
When the split is larger (2 or 3), the min effect is reduced to 16 msec. 
Finally, with a split of 4, there is no min effect, A similar analysis of 
the Buckley and Gillman (1974) dot comparison data, based on a larger range 
of roins and splits, further siqjports this conclusion.^ For splits of 1, 2, 
3, 4, 5, 6, and 7» the min effects are 66,57 49, 34, 25, 16, and 8 msec, 
respectively* In other words, there is a clear monotonic decrease in the 
min effect as the split increases. Also, there is a main effect of the 
split such that the latency generally decreases as the split increases. The 
important point, as far as the two-process explanation is concerned, is that 
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Table 2 

Mean Latencies in msec (and % error) 
for the IS Different Pairs of. Group Sizes 

Number of Dots in Smaller Group 
1 2 3 4 S 

2 817 (4.0) 

3 7.S7 (2.8) 844 (1.7) 

4 718 (2.3) 744 (2.3) 873 (4.S) 

5 730 (2.3) 729 (1.1) 770 (1.1) 916 (7.4) 

6 716 (2.3) 731 (3.4) 750 (1.7) 800 (1.7) 997 (4.0) 

MEAN 748 762 798 858 997 
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when the split is larger, the category judgment mechanism may be used more 
often ^ and so the min effect decreases. 

Other types of explanations account for the quantitative contparison 
task in terms of a quasi-logarithmic analogue representation of quantities 
(Buckley g Gillman, 1974; Moyer g Landauer, 1967} Shepard, Kilpatric^ § 
Cunningham, 1975)* The advantage of these alternative explanations is that 
they are parsimonious, and they seem readily applicable to continuous dimen- 
sions, such as sizes of animals (Moyer, 1973). However^ these approaches 
can not easily account for the finding that the gaze duration on both groups 
of dots was proportional to the size of the smaller group. By contrast^ a 
counting model is easily compatible with this aspect of the data. 

The duration of gaze on the sector other than the larger and smaller 
group of dots did not vary from condition to condition and showed little 
evidence of a min effect or a split effect. Incorrect responses were rare 
(2.8%), and the errors were distributed as shown in Table 2. The mean 
processing time in this task, 793 msec, was distributed as follows. On 
average, 371 msec were spent gazing at the word more or less, 178 msec 
gazing at the larger group of dots, 165 at the smaller group^ and 79 msec 
at the sector between the two groups. The eye fixation data also showed 
Kow this distribution of processing time was affected by the sizes of the 
smaller and larger groups. The results indicated that the two groups of 
dots are fixated in a manner consistent with an upward counting process, 

OVERVIEW 

The locus of the fixation . The most general assumption of the current 
research is that locus of the eye fixation can indicate what symbol is cur- 
rently being processed. Converging lines of evidence from very diverse tasks 
support this gcncrai h>7>othcr.i;; and also allow us to refine our theoretical 
consideration about the relationship between eye fixations and mental proces- 
ses. 
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In tasks where the behavioral units are fairly large and open to con- 
scious introspection, the pattern of eye fixations correlates well with sub* 
jects' verbal reports. For example, Winikoff (1967, see also Newell 6 Simon, 
1972) found a correlation between eye fixations and verbal report in crypta- 
rithmetic tasks, where numbers are substituted for letters to solve a problem 
like DONALD GERALD - ROBERT. The aggregation rules cumulated many fixations 
(as much as S sec of activity) into a typical processing unit* In general, 
Winikoff's stAject tended to look at the letter whose value he was computing 
or trying to recall, as inferred from his concomitant verbal protocol*- Sim* 
ilarly^ eye fixations correlate with verbal protocols when subjects are 
choosing among several alternatives such as cars that differ in make, year^ 
and condition (Russo 6 Rosen, 197S) * These studies provide evidence that 
the locus of the eye fixation corresponds to the information being processed 
in tasks where subjects can verbalize what they are processing* 

Some aspects of problem solving involve operations too rapid for ver- 
bal protocols, but the eye fixations still reveal what symbols the subjects * 
are processing* A good example are the few ^can paths that have been recor- 
ded of chess masters scanning a board position for S sec (de Groot § Jong- 
man, 1973; Tichomirov 5 Posnyanskaya, 1966)* The locus o£ eye fixations is 
accounted for by assuming tliat the master scans between pairs of pieces that 

are related by attaclc or defense (Simon 5 Barenfeld, 1969)* Again, these 

* 

data support the hypothesis that the locus of the eye fixations reflects 
what is being internally processed* 

Since eye fixations are sensitive to the structure of the internal re- 
presentation being constructed or operated upon, they provide a valuable 
methodology for examining how linguistic material is interpreted* Our own 
research stratepy in tlii?; aroa has boon to present a linguistic stimulus^ 
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followed by a picture, and examine how the internal representation of the 
prior sentence alters the way the picture is scanned in a verification task. 
For example, we have used this methodology to examine the processing of 
affirmative and implicitly negative sentences (Carpenter § Just, 1972), 
The affirmative sentences (e.g., A small proportion of the dots are red) 
and the implicitly negative sentences (e.g,. Pew of the dots are red] have 
the same truth value. However, linguistic and psychological evidence sug* 
gests that the two sentences have different internal representations 
(Just § Carpenter, 1971). The affirmative sentence is represented as an 
affirmation that the small subset has some property, in this case, redness. 
We predicted that after reading the affirmative that refers to the small 
subset (e.g,, A small prop^^rtion , , J , people should tend to fixate the 
small subset. By contrast, an implicit negative is represented as a negation 
of some property of the large subset, in thii case, redness. We predicted 
that after reading an implicit negation about the large subset (e.g.^ Few of 
the dots are red) , people would tend to fixate the large subset. As predic- 
ted, subjects looked at the location in the picture specified by the under- 
lying representation of the sentence. The locus of the eye fixation Is 
sensitive to the deep structure representation, even when subjects aren*t 
consciously aware of the nature of the linguistic stimulus or of their pattern 
of eye fixation. 

While people are listening to spoken questions or passages, they tend to 
fixate the pictorial referent of words that occur in the text (Cooper^ 1974j 
Kahneman 6 Lass, 1971, cited by Kahneman, 1973), For example^t in the 
Kahneman and Lass study, people 
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were shown a schematic drawing of four objects, such as a car, person, tree 
and airplane and asked a question like "What makes of cars can you name?" 
Subjects tended to look at the schematic car while answering. More interes- 
tingly, when the picture was removed prior to the question, subjects still 
tended to look where the appropriate object had been located* Such fixations 
apparently play a place-keeping organizational role rather than an encoding 
role. The symbols in the short term memory may be indexed to particular 
spatial locations. (This formulation is reminiscent of the method of loci 
(cf. Bower, 1970) and spatial interference effects in retrieval (Byrne,- 1974)). 
When the time comes to retrieve or operate on a symbol, the eye may fixate 
the location from vrfiich the symbol was originally encoded. It may be this 
mechanism that produces fixations on the referent of the symbol at the top 
of the stack, assuming that the referent stays in the same location. 

Duration of gaze . In the tasks we investigated, the time spent gazing 
at a figure reflected both the time to encode that figure as well as the 
time to operate on the encoded symbol. Tachistoscopic recognition studies 
indicate that familiar figures, like alphanumeric characters or even words 
can be internalized within a very short exposure duration — as low as a few 
tens of milliseconds. Yet in these cognitive tasks, people gaze at very 
simple and familiar figures for much longer, often for hundreds of millise- 
conds. For example, in the sentence verification task, subjects looked at 
a star (*) or a plus (+) for 700 to 1200 msec, depending upon the relation 
between the sentence and the figure. Clearly, the duration of the gaze in-, 
eludes not only encoding time but also the time for subsequent operation on 
the encoded symbol. 

There are a number of reasons why a subject might continue to fixate a 
figure after the relevant information has been encoded. If the processor 
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is busy operating on the most recently encoded information, there is no 
reason for it to direct the eye to seek other information* So the eye may 
remain stationary simply because it is not instructed to move* An alterna- 
tive view of the persistence of the gaze is that the processor might active- 
ly instruct the eye not to move during the processing of the most recently 
encoded information. The reason for avoiding new fixations might be that a 
saccade automatically initiates an encoding activity (cf. Loftus, in press) 
that could interrupt the ongoing processing* Perhaps the reason that people 
often gaze upwards or close their eyes altogether while computing the answer 
to a demanding question is that they are avoiding extraneous encoding oper- 
ations that could interrupt processing* Thus, the persistence of the gaze 
could be due to the absence of an instruction to move the eye or the presence 
o£ an instruction not to move the eye* In either case, the gaze duration 
on a particular figure provides a measure of the time spent processing the 
corresponding symbol. 

One of the most elegant studies of the relationship between gaze dura- 
tion and mental operation examined gaze duration in a Sternberg memory-scan- 
ning task (Gould, 1973). In Gould*s experiment, subjects had a memory set 
of 1, 2, or 3 letters, and 12 probe letters were distributed along the peri- 
meter of an imaginary clock face that corresponded to the display* Only one 
of the 12 probe letters was a member of the memory set* The subject's task 
was to scan around the clock face (starting at 12 o* clock and proceeding 
clockwise) until he found the positive probe* 

The duration of the time spent fixating each negative probe item in^ 
creasea linearly with the memory set size, at a rate of about SO msec/item* 
This is compatible with the explanation that each probe item was serially 
compared to each of the memory set items, and as the memory set size increased, 
the probe had to bo compared to more items in memory* The eye fixated the 
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probe while the comparison operations occurred* The importance of this fin* 
ding is that the parameter of SO msec per item, inferred from the duration 
of the gase, is very close to the time of 38 msec per item inferred from re- 
action time studies with only a single probe (Sternberg, 1969). The duration 
of fixation on the positive probe item also Increased as a function of the 
memory set size. Of particular importance is that the rate of increase in 
this duration, the slope, was the same for the positive probe as for the 
negative probes. This result suggests that the scan through the memory set 
is exhaustive. 

Gould's results provide an important validation for the eye fixation 
methodology. The eye fixation measures yield results that are completely 
consistent with Sternberg's careful reaction-time studies. Gould's study* 
as well as the current research, combines the use of additive factors tech- 
niques and eye fixation measures to estimate the time taken to execute a 
mental operation. 

The only eye fixation research that reports a lack of correlation be- 
tween fixation duration and performance concerns memory for pictures (tof- 
tus, 1972). Loftus found that during learning, the number of fixations, not 
their total duration, was the best predictor of subsequent recognition memo- 
ry. However, Tversky (1974) has recently found a positive correlation be- 
tween the duration of individual eye fixations and later memory. Tversky 
suggests that the critical variable in relating fixation duration to picture 
memory might be the kind of features being encoded in the learning phase. 
The present research does not attempt to account for eye fixations in pic- 
ture scanning and recognition (for relevant work on this topic, see Buswell^ 
1935; MackwoTth 6 Bruner, 1970; Mackworth g Morandi, 1967; Noton § Stark, 
1971; Potter Q Levy, 1969). Our concern has been with ongoing computation 
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rather than search processes in long-term memory, so the picture recognition 
issue would take our discussion too far afield. However, the present theo- 
retical framework suggests that the resolution of this problem requires a 
model of what is encoded during the initial learning and what tests are 
made during the subsequent recognition phase. As yet, the data relating 
fixation duration to recognition memory are insufficient to construct a com- 
plete model of the mental processes in picture memorization and retrieval* 

We have related mental operations to the duration of gaze, not neces- 
sarily to the duration of individual fixations. Fixations must be aggre- 
gated into a unit consistent with the underlying theory. For example, in 
the sentence verification task, when a person looked at the plus, whether 
he made one or two fixations was irrelevant to the fact that he was looking 
at that piece of information and no other. We measured how long he looked 
at the plus, not how many fixations actually went into that gaze time. The 
duration of gaze was systematically related to the duration of underlying 
processes. The duration of individual fixations may also be so related, but 
only under those conditions in which each fixation is a psychologically sep- 
arate unit (cf. also Carpenter 6 Just, in press; Russo S Rosen, 1975). 

Task conditions that ^p timige_the use of eye fixations . The locus of 
fixation is not always synonymous with the direction of attention. Subjects 
can be instructed to fixate one referent while attending elsewhere. The 
possibility of such disassociation makes it important to specify the condi^ ' 
tions under which eye fixations are an accurate reflection of what is being 
processed. Ore of the most important conditions is that the task require 
that information from the visual environuient be encoded and processed* If 
the visual display is not relevant, there are no mapping rules between what 
is being fixated and whnt is being internally processed. A second condition 
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is that the task goals be specified for the subject. Asking subjects simply 
to look at a picture or read some prose permits them to adopt their own def- 
initions of what processing is required and this again makes it difficult 
to infer the relationship between eye fixations and underlying mental pro- 
cesses. And of course^ speeded tasks discourage extraneous processing and 
the concomitant extraneous fixations. 

Some of the rules that govern fixations are general scanning strategies ^ 
while other rules are highly specific to the processing in the task being 
performed. Eye fixations will reveal the mental processes in a particular 
task only if the task structure minimizes the use of general scanning stra- 
tegies. An example of this structuring is evident in the study of how peo- 
ple looked at pictures after reading sentences involving affirmative quantifiers 
e.g.^ A small proportion of the dots are red^ or negative quantifiers^ 
e*g.. Few of the dots are red (Carpenter § Just, 1972). The pictures always 
had a small subset of dots at the top and a large subset at the bottom. 
Thus, the subject knew to look at the top or at the bottom, depending on 
whether he wanted to determine the color of the small subset or the large 
one. This task structure eliminated the need first to search for the de« 
sired subset and then to encode its color. The relation between eye fixations 
and mental operations is even clearer when the role of peripheral informa- 
tion is controlled. The extreme case of this is the computerized "tunnel 
vision" in the sentence verification task^ in which there is no peripheral 
information, so the duration of gaze at any locus cannot reflect encoding of 
information from another locus. These features of the task structure mini- 
mize the role of general scanning strategies and thereby make the design, 
more sensitive to the cognitive processes of interest. 

In a)l of these tasks, the eye scan is very miich Roal directed, in 
fact^ directed by the information present "at the top of the stack." There 
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are two possible sources of such information, namely, the task structure and 
information computed during the trial. Both sources influenced fixations 
in the sentence verification task where the instructions to fixate the sen- 
tence determined the first fixation, but the locus of the second fixation 
was determined by information computed during the trial* After the sentence 
(e.g* Plus isnU North) was fixated, the directional term in the sentence 
determined the locus of the next fixation, in this case. North. Since botli 
the task structure and the ongoing processing can determine the locus of 
fixation, both factors must be taken into account in developing a complete 
processing model. 

One domain of eye fixation research that has been hampered by the ab- 
sence of task analyses is the area of reading. While there have been many 
promising empirical studies of eye fixations in reading (cf. Buswell, 1922, 
1937; Hochberg, 1970} Kolers, 1970; Levin 6 Kaplan, 1970; Mackworth, 1974; 
McConkie § Rayner, 1974; Mehler, Sever, g Carey, 1967; Tinker, 1958) * there 
is no convergence on a theory of reading* The difficulty is that there is 
no single "reading process", because we read differently in different sit- 
uations. For example, a newspaper article is read differently from a legal 
contract, and the same contract is read differently depending on Whether 
ohe is looking for typographical errors or buying a house. One possible 
solution to this difficulty is to systematically examine a number of well- 
defined reading tasks that are of inherent interest. For example, we are 
currently examining eye fixations of subjects who read and solve three term 
series problems (e.g.. If John is leading Bill^ and Tom is following Bill ^ 
then where is John? ) , In another study, we are examining subjects* eye fix- 
ations as they read a passage and answer questions about it in a standar- 
dized reading conprcticiusion test* In order to de\'elop models of reading, 
it will probably be necessary to study performance in a number of well-under- 
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stood task environments, so as to determine the influence of the environ- 
ments on the reading process. 

Generalization of the models . Certain kinds of operations in the cen- 
tral processor appear to function similarly irrespective of the source of en- 
coding of the operated-on symbol, be it a visual display, tactile input, 
semantic memory retrieval or whatever. In those cases in which the operations 
are invariant, conclusions gained from the eye fixation methodology may gen- 
eralize to processing of symbols in non-visual domains. The invariant op- 
erations would presumably be very basic ones, such as comparing two symbols 
for identity, retrieving the next symbol in an ordered list, or incrementing 
an internal counter. 

One example from recent psycholinguistic research demonstrates how sen- 
tences that refer to information from different sources (like pictures vs 
semantic memory) may be processed similarly. Just (1974) timed subjects 
vtfiile they verified quantified sentences like Some of the red figures are 
round with respect to a picture that included red and round figures* The 
overall pattern of latencies was similar to the pattern obtained when the sen- 
tences refer to concepts in semantic memory, e.g. Some men are doctors (Mey- 
er, 1970)* In fact, even though the relevant information was encoded from 
a picture in one case and retrieved from semantic memory in the other, both 
sets of data could be explained in terms of the same operations (Just, 1974). 
Obviously, the initial encoding stages involve different processes, but in 
this and certain other cases (cf. Carpenter ^ Just, 1975), the information 
seems to be manipulated similarly once it is past the encoding stages. This 
suggests that processinjg models of these subsequent stages derived from eye 
fixations studies may generalize to non*visual domains. 

Internal rotation processes may also be somewhot independent of the vi- 
sual modality. Wlien subjects are deciding whether a visually presented^t ro- 
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tated '*R" is normal or a mirror image (Cooper tt Shepard, 1973), the response 
latencies resemble those for the Shepard and Metzler task in certain re- 
spects. The resemblance led Cooper and Shepard to argue that the processes 
in the two tasks were similar. In the Cooper and Shepard task there cannot 
be eye fixations switching back and forth between the two R's, since only 
one of them is externally present, while the other is the long term repre- 
sentation of a normal R. Nevertheless, it is reasonable to speculate that 
the sequence of internal switches of attention in the Cooper and Shepard 
study is related to the external sequence of fixation switches observed in 
our study of rotation. If this speculation is correct, the model we have 
proposed may have some application to rotation processes that involve men- 
tally generated stimuli- 

If processing models based on eye fixation studies are to be generalized 
to non-visual tasks then the factors that influence only visual encoding 
must be identified- For example, picture scanning processes might be affec- 
ted by perceptual saliency (Williams, 1966), and there may be no parallel 
in semantic memory retrieval. Conversely, semantic memory retrieval may be 
affected by factors such as semantic distance (cf* Rips, Shoben § Smith, 1973), 
which has no parallel factor in picture encoding processes* If these modal- 
ity-specific processes can.be isolated, then eye fixations may provide a way 
to investigate the fundajnental operations that occur in the central processor* 
Operations whose durations lie between SO and 800 msec seem especially sus- 
ceptible to this approach, as showii by the current work on rotation, sentence 
verification, rnd quantitative comparison* For these rapid operations, there 
is a very close link between the symbol that is being processed and the locus, 
sequence and duration of eye fixations, because of the eyes* tendency to fix- 
ate the referent of tJie symbol t\mt is " at the top of the stack". 
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FOOTNOTES 



The order of authors is arbitrary. This paper represents a collaborative 
effort. 
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2 

Metzler and Shepard (1974) report some preliminary observations on the eye 
movements of two subjects performing the mental rotation task; their subjects 
also looked back and forth between the two figures* 

The SO" steps indicated by our data are suggestively close to 45*, which 
has more intuitive appeal. 

4 

The analysis is based on cell means estimated from a graph of the latencies 
for comparing random configurations of dots (Gillman § Buckley^ 1973). The 
aggregated data appear in Buckley and Gillman (1974). 
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